Papers with AI development
H2O Open Ecosystem for State-of-the-art Large Language Models (2023.emnlp-demo)
Copied to clipboard
Arno Candel, Jon McKinney, Philipp Singer, Pascal Pfeiffer, Maximilian Jeblick, Chun Ming Lee, Marcos Conde
| Challenge: | Large Language Models (LLMs) are a revolution in AI, but they pose many significant risks, such as the presence of biased, private, copyrighted or harmful text. |
| Approach: | They propose to develop and test Large Language Models using open-source tools and frameworks. |
| Outcome: | The proposed framework and models are licensed under Apache 2.0 licenses. |
SynthTextEval: Synthetic Text Data Generation and Evaluation for High-Stakes Domains (2025.emnlp-demos)
Copied to clipboard
Krithika Ramesh, Daniel Smolyak, Zihao Zhao, Nupoor Gandhi, Ritu Agarwal, Margrét V. Bjarnadóttir, Anjalie Field
| Challenge: | SynthTextEval is a toolkit for conducting comprehensive evaluations of synthetic text. |
| Approach: | They propose a toolkit for conducting comprehensive evaluations of synthetic text using large language models. |
| Outcome: | The proposed toolkit can be run over any dataset, but it is aimed at two high-stakes domains: healthcare and law. |
Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignment (2026.acl-srw)
Copied to clipboard
| Challenge: | Disagreement arises from subjective human opinion and can vary with one’s identity, beliefs, and social environment. |
| Approach: | They propose a human-centered framework for reproducible ML evaluation and AI alignment that takes disagreement into account when building human-centric AI systems. |
| Outcome: | The proposed framework is based on a human-centered and perspective-aware framework for reproducible ML evaluation and AI alignment. |
Multi-Hall-SA: A Cross-lingual Benchmark for Multi-Type Hallucination Detection in Low-Resource South African Languages (2026.findings-eacl)
Copied to clipboard
| Challenge: | Large Language Models generate false or unsupported information, which can be difficult to detect in low-resource languages. |
| Approach: | They propose a cross-lingual benchmark for hallucination detection spanning English and South African languages. |
| Outcome: | The proposed model detects 23.6% fewer hallucinations in South African languages compared to English . human validation confirms the quality and cross-lingual alignment of the model . |
Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that language model benchmarks are vulnerable to manipulation and exploitation. |
| Approach: | They propose a method that allows the covert transfer of benchmark-specific knowledge through seemingly legitimate intermediate training steps. |
| Outcome: | The proposed method can achieve significant improvements in accuracy without developing reasoning capabilities. |
Defining and Evaluating Visual Language Models’ Basic Spatial Abilities: A Perspective from Psychometrics (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies assessing the spatial abilities of VLMs lack a solid theoretical foundation and lack measurable data. |
| Approach: | They propose a psychometric framework defining five basic spatial abilities in Visual Language Models. |
| Outcome: | The proposed framework defines five basic spatial abilities in Visual Language Models (VLMs) it provides a comprehensive evaluation benchmark and methodological perspective for embodied AI development . |
Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMs (2025.findings-emnlp)
Copied to clipboard
Kuan Lok Zhou, Jiayi Chen, Siddharth Suresh, Reuben Narad, Timothy T. Rogers, Lalit K Jain, Robert D Nowak, Bob Mankoff, Jifan Zhang
| Challenge: | Large Language Models (LLMs) have shown significant limitations in understanding creative content, as demonstrated by Hessel et al. (2023)’s influential work on the New Yorker Cartoon Caption Contest. |
| Approach: | They propose to decompose humor understanding into three components and improve each by enhancing visual understanding through improved annotation and utilizing LLM-generated humor reasoning and explanations. |
| Outcome: | The proposed approach achieves 82.4% accuracy in caption ranking, significantly better than the previous 67% benchmark and matches the performance of world-renowned human experts in this domain. |
BenNumEval: A Benchmark to Assess LLMs’ Numerical Reasoning Capabilities in Bengali (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel in general-purpose tasks but struggle with numerical reasoning, especially in low-resource languages like Bengali. |
| Approach: | They propose a benchmark to assess LLMs on numerical reasoning tasks in Bengali. |
| Outcome: | The proposed benchmark assesses LLMs on numerical reasoning tasks in Bengali. |
Data Pollination: An Emergent Ecological Process Driving AI Population Evolution (2026.acl-long)
Copied to clipboard
| Challenge: | evidence from deployed systems suggests that language models interact through a shared data ecosystem. |
| Approach: | They propose to use data pollination to investigate stability dynamics under synthetic data training to investigate model collapse. |
| Outcome: | The proposed model can mitigate model collapse observed in recursive training, and improve performance across benchmarks. |
Beyond A Single AI Cluster: A Survey of Decentralized LLM Training (2025.emnlp-main)
Copied to clipboard
| Challenge: | Decentralized LLM training leverages dispersed resources at varying scales. |
| Approach: | They propose a resource-driven paradigm that leverages dispersed resources across clusters, datacenters and even regions. |
| Outcome: | The proposed model scales are 175 billion to 660 billion parameters, and the exponential growth in computational requirements poses significant challenges. |